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Command  and  Control  Team  Performance  in 
Distributed  Mission  Training  Exercises 


Introduction 


The  use  of  realistic  simulation  equipment  has  been  standard  practice  in  military  operational 
training  for  quite  some  time.  Distributed  Mission  Training  (DMT)  takes  this  concept  further,  by  creating  a 
synthetic  battlespace  with  multiple  participants.  This  is  accomplished  through  networking  of  simulators 
to  enable  a  complex  battlespace  environment  filled  with  live  (human  in  aircraft),  virtual  (human  in 
simulation),  and  constructed  (computer-driven)  forces.  It  is  expected  to  reduce  the  number  of  flight 
hours  while  maintaining  levels  of  expeditionary  force  readiness.  DMT  is  looked  upon  as  a  way  that  will 
allow  warfighters  to  train  in  a  manner  consistent  with  Air  Force  Doctrine  60:  Train  like  you  fight,  fight  like 
you  train.  Previous  constraints  (e.g.,  safety,  flying  status,  flying  hours,  and  aircraft  and  environmental 
factors)  have  hampered  training  efforts  from  following  this  philosophy  to  its  fullest  capacity.  DMT  is 
consistent  with  the  vision  explicated  by  General  Fogleman,  Chief  of  Staff  Air  Force,  on  29  Jan  1996: 

I  am  convinced  modeling  and  simulation  technologies  available  today  will  enable  us 
to  significantly  change  the  way  we  train  in  the  future.  We  are  at  a  crossroads  where 
simulator  technology  today  will  be  critical  in  the  success  of  our  effective  use  of  follow-  . 
on  weapon  systems  . . ..  We  need  to  take  a  hard  look  at  how  this  technology  will 
change  our  training  philosophy  as  well  as  how  we  develop  future  weapon  systems. 

DMT  offers  flexibility  that  stretches  worldwide  by  linking  participants  across  network  lines, 
allowing  individual  warfighters  to  train  concurrently  within  a  shared  mission  scenario.  This  capability 
supplements  aircrew  flight  hours  with  a  synthetic  reproduction  of  a  complete  and  complex  warfighting 
environment  that  combines  realistic  scenarios  with  simulated  operational  systems.  It  allows  individuals  to 
experience  the  full  extent  of  interdependencies  among  various  players  and  to  participate  in  debriefing 
sessions  that  may  not  occur  in  live-fire  exercises  or  in  combat  situations.  For  example,  the  Airborne 
Warning  and  Control  System  (AWACS)  crew  do  not  normally  debrief  with  the  pilots  of  fighter  aircraft,  due 
to  the  extended  time  the  AWACS  crew  are  in  flight.  Thus,  blame  for  errors  can  easily  occur  without 
resolution,  due  to  lack  of  debrief  communication. 

The  ROADRUNNER  ‘98  exercise  focused  primarily  on  the  integration  of  multiple  warfighter-in- 
the-loop  training  simulators,  combining  them  with  synthetic  forces  to  form  a  virtual  battlespace  to  conduct 
several  assorted  combat-oriented,  tactical  training  exercises.  ROADRUNNER  '98  was  sponsored  by  the 
Air  Force  Modeling  and  Simulation  Office  (USAF/XOC)  with  the  Air  Force  Research  Laboratory,  Human 
Effectiveness  Directorate’s  Warfighter  Training  Research  Division  (AFRL/HEA)  serving  as  program 
managers  and  the  Theater  Command  and  Control  Simulation  Facility  (TACCSF)  as  systems  integrators. 
Extensive  support  was  received  from  the  Training  Office  at  Air  Combat  Command’s  Deputy  for 
Operations  (ACC/DOT),  the  Air  Force  Information  Warfare  Center’s  Advanced  Combat  Simulations 
Division  (AFIWC/SAM),  the  AWACS  training  office  at  the  522nd  Air  Control  Wing  (ACW),  the  107th  Air 
Control  Squadron  (ACS)  of  the  Iowa  Air  National  Guard  (IANG),  and  the  Naval  Air  Warfare  Center’s 
Training  Systems  Division  (NAWC-TSD).  The  warfighters  who  participated  in  ROADRUNNER  ‘98  were 
operational  pilots,  weapons  directors  (WDs),  and  air  surveillance  technicians  (ASTs)  from  the  522nd  ACW, 
Tinker  AFB,  OK;  27th  Fighter  Wing,  Cannon  AFB,  NM;  the  185th  and  132nd  Fighter  Wings  of  the  IANG;  33rd 
Fighter  Wing,  Eglin  AFB,  FL;  and  from  the  United  States  Air  Forces  in  Europe  (USAFE). 

The  overall  goal  of  this  effort  was  to  compare  initial  team  performance  with  subsequent 
performance  after  participation,  and  give  the  participants  the  opportunity  to  critique  both  the  systems  and 
the  concept  of  training  (Parton,  1998).  Each  scenario  was  based  on  intelligence  briefings  and  air  tasking 
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order  (ATO),  and  each  fighter  and  AWACS  team  planned,  coordinated,  and  briefed  their  mission.  The 
combined  teams  flew  the  mission  in  their  respective  simulators  with  a  “trusted  agent”  monitoring  and 
evaluating  their  performances.  These  trusted  agents  were  AWACS  WD  instructors  who  assisted  in 
development  of  evaluation  forms  and  served  as  experts  for  the  collection  of  observed  data.  The  teams 
and  instructors  participated  in  an  after-action  review  aided  by  a  mission  playback  system  that  assisted 
during  the  debriefing  process.  Each  team  participated  in  seven  separate  missions  over  the  course  of  one 
week. 


While  DMT  offers  opportunities  for  the  investigation  of  performance  within  highly  complex  and 
interdependent  scenarios,  it  also  presents  additional  challenges  with  regard  to  identification  of  training 
requirements,  scenario  development,  and  performance  evaluation.  One  challenge  arises  when  scenarios 
are  expected  to  provide  valuable  training  experience  to  all  participants.  For  example,  a  scenario  that  is 
challenging  and  relevant  to  the  training  requirements  of  fighter  aircrew  may  not  produce  any  challenge  for 
other  participants.  Additional  effort  must  be  directed  to  the  creation  of  a  scenario  that  provides  valuable 
experience  for  all  participants,  and  the  utilization  of  constructed  forces  (computer-based  entities  driven  by 
intelligent  agent  technology)  to  provide  supporting  roles. 

Another  challenge  inherent  in  DMT  scenarios  is  the  measurement  of  performance  of  individual 
participants,  of  teams,  and  of  the  overall  mission.  Dwyer,  Oser,  Salas,  and  Fowlkes  (1999)  have 
discussed  the  challenges  to  performance  measurement  in  distributed  training  environments.  These 
include  the  geographic  separation  of  players  and  the  rich  complexity  of  tasks  and  their  interactions.  They 
provide  sound  guidelines  for  an  approach  to  performance  measurement  that  is  linked  to  training 
objectives  and  scenario  events,  referred  to  as  event-based  training  (EBT). 

In  this  effort,  we  utilized  this  EBT  approach  for  scenario  development  to  refine  the  AWACS 
portion  of  the  scenario.  Our  first  challenge  lay  in  the  fact  that  the  AWACS  mission  was  more  of  a 
supportive  role  to  that  of  the  fighter  aircraft  performing  in  Mesa,  Arizona.  We  had  to  enrich  the  AWACS 
roles,  identify  training  requirements,  and  create  trigger  events  to  elicit  and  evaluate  individual  and  team 
performance. 

At  the  same  time,  we  wanted  to  focus  on  teamwork  aspects  of  performance.  Thus,  we  used  this 
opportunity  to  further  refine  a  taxonomy  of  command  and  control  team  performance,  through  focal  group 
interviews  of  AWACS  WD  instructors.  From  this  taxonomy,  we  generated  a  rating  form  that  was  focused 
on  teamwork  aspects  of  performance.  Thus,  our  scenarios  were  refined  through  EBT  procedures,  and 
our  evaluation  measures  were  partially  EBT-driven,  but  supplemented  by  a  more  construct-based 
approach  to  performance  measurement.  We  wanted  to  create  a  taxonomy  of  command,  control,  and 
communication  (C3)  performance  which  would  be  generalizable  to  other  interventions  in  addition  to  that 
of  training.  For  example,  in  future  studies  we  shall  look  more  closely  at  the  impact  of  factors  such  as  the 
distribution  and  display  of  information. 

Command  and  Control  Teams 


All  command  and  control  teams  share  some  common  task  and  performance  characteristics. 
These  teams  perform  in  highly  .interdependent  roles,  tracking  and  coordinating  some  type  of  tactical 
action,  in  a  manner  consistent  with  overall  strategic  goals  and  procedures,  for  a  defined  sector  of  air 
and/or  land  space  over  a  sustained  period  of  time. 

The  AWACS  team  serves  within  this  performance  domain,  to  augment  the  need  for  airborne 
surveillance  and  command,  control,  and  communications  functions  for  tactical  and  air  defense  forces. 
They  provide  the  means  to  detect,  identify,  track,  and  intercept  airborne  threats.  The  aircraft’s  god’s-eye 
capability  offers  an  altitude-independent  360°  view  of  more  than  200  miles  over  both  land  and  water. 
There  are  five  mission  crew  categories  related  to  the  function  of  the  AWACS  crew:  the  Mission  Crew 
Commander  (MCC),  Senior  Director  (SD),  Weapons  Director  (WD),  Air  Surveillance  Officer  (ASO),  and 
Air  Surveillance  Technician  (AST).  AWACS  teams  typically  work  in  teams  of  two  to  four  crewmembers  as 
part  of  the  airborne  command  and  control  group  located  within  E-3  Sentry  aircraft. 
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The  AWACS  crew  coordinates  communications  received  from  a  number  of  sources,  such  as 
other  WDs,  the  SD,  air  surveillance  personnel,  electronic  combat  officers  (ECOs),  intelligence  operations, 
base  operations,  and  friendly  pilots.  To  accomplish  this,  they  must  exchange,  interpret  and  effectively 
weight  information  and  optimize  resource  allocation  decisions  across  team  members,  over  time,  and 
under  stress  and  fatigue.  These  decisions  depend  on  sharing  resources,  such  as  surface-to-air  missile 
sites  and  various  combat,  reconnaissance,  refueling,  and  search  and  rescue  aircraft.  Relevant 
information  must  be  distributed  to  appropriate  personnel  and  updated  over  time,  in  dynamic  conditions 
which  may  require  shift  changes  in  personnel.  Information  is  often  verbal,  and  may  be  missing, 
degraded,  passed  along  from  unfamiliar  sources,  or  misinterpreted  by  others.  In  addition,  information  is 
often  communicated/interpreted  by  individuals  with  only  partially  overlapping  awareness  of  the 
battlespace. 

Cognitive  Task  Analyses.  For  the  ROADRUNNER  ’98  study,  assessment  of  AWACS  team  of 
performance  was  based  on  subject-matter  expert  (SME)  ratings  of  several  dimensions.  These 
dimensions  were  generated  based  on  existing  taxonomies  of  team  performance,  and  cognitive  task 
analyses  of  AWACS  performance,  followed  by  focus  groups’  interviews  of  experienced  AWACS 
instructors.  These  focus  groups  also  served  to  refine  the  rating  form,  and  included  the  SMEs  who 
actually  participated  as  ROADRUNNER  ’98  trusted  agents/evaluators. 

'For  this  effort,  cognitive  task  analyses  informed  and  influenced  the  development  of  scenario 
characteristics  and  the  identification  of  primary  performance  constructs  and  measures.  Several  cognitive 
task  analyses  have  been  performed  on  AWACS  operational  personnel.  First,  analyses  have  been 
performed  with  regard  to  AWACS  individual  tasks,  with  a  primary  focus  on  display  enhancements  (Klinger 
et  al.,  1993).  In  addition,  Fahey  and  associates  (Fahey  et  al.,  1998)  investigated  the  AWACS  task 
domain  with  a  special  focus  on  AWACS  team  tasks,  utilizing  cognitive  task  analysis  and  critical  incident 
techniques  for  data  elicitation.  Their  report  provides  a  comprehensive  description  of  the  AWACS  task, 
and  the  general  finding  that  critical  incident  technique  was  quite  useful  for  the  description  of  AWACS 
teamwork.  In  addition,  a  team-based  analysis  of  AWACS  tasks  was  performed  (MacMillan  et  al.,  1998). 
Results  were  very  informative,  supporting  frameworks  and  constructs  within  an  information-requirements 
approach  to  task  descriptions.  . 

Identification  of  C3  Teamwork  Constructs 

The  starting  point  to  identify  C3  teamwork  constructs  began  in  our  search  for  core  characteristics 
of  teamwork  performance  in  general.  We  reviewed  existing  taxonomies  and  identified  several  compelling 
alternatives  (Baker  &  Salas,  1992;  Cannon-Bowers,  Oser,  &  Flanagan,  1992;  Fleishman  &  Zaccaro, 
1992;  Hackman  &  Morris,  1975;  Kozlowski,  1998;  O’Neil,  Baker,  &  Kazlauskas,  1992;  Smith-Jentsch, 
Johnston,  &  Payne,  1998;  Sundstrom,  De  Meuse,  &  Futrell,  1.990;  Swezey,  Meltzer  &  Salas,  1994).  It  is 
evident  that  teams  span  a  diverse  array  of  functions.  Several  taxonomies  have  provided  some  key 
distinguishing  factors  among  these  teams  (Salas,  Bowers,  &  Cannon-Bowers,  1995;  Salas,  Dickinson, 
Converse,  &  Tannenbaum,  1992;  Sundstrom,  et  al.,  1990;  Swezey,  et  al.,  1994).  Others  have  focused  on 
components  relevant  to  teamwork  training  and  adaptive  expert  performance  (Andrews,  Waag,  &  Bell, 
1992;  Baldwin  &  Magjuka,  1997;Blickensderfer,  Cannon-Bowers,  &  Salas,  1998;  Bowers,  Baker,  & 
Salas,  1994;  Eggemeier,  Fisk,  Robbins,  &  Lawless,  1988;  Ford,  Kozlowski,  Kraiger,  Salas,  &  Teachout, 
1997;  Goldsmith  &  Kraiger,  1997;  Kozlowski,  1998;  Kozlowski,  Gully,  McHugh,  Salas,  &  Cannon-Bowers, 
1996;  Kozlowski,  &  Salas,  1997;  Kozlowski,  Gully,  Nason,  &  Smith,  1999;  Kraiger,  Ford,  &  Salas,  1993; 
Mathieu  &  Martineau,  1997;  Mathieu,  Tannenbaum,  &  Salas,  1997;  McIntyre  &  Salas,  1995;  Salas, 
Cannon-Bowers,  &  Blickensderfer,  1997;  Salas,  Cannon-Bowers,  &  Johnston,;  1997;  Salas,  Cannon- 
Bowers,  &  Kozlowski,  1997;  Smith,  Ford,  &  Kozlowski,  1997;  Serfaty,  Entin  &  Johnston,  1998;  Smith- 
Jentsch,  Johnston,  &  Payne,  1998;  Smith-Jentsch,  Zeisig,  Acton,  &  McPherson,  1998;  Swezey  &  Salas, 
1992;  Tannenbaum,  Smith-Jentsch,  &  Behson,  1998). 

Each  taxonomy  was  useful  for  its  particular  domain  and  purpose,  however,  we  were  searching  for 
a  taxonomy  that  could  possibly  distinguish  teams  in  general.  Constructs  in  some  of  the  taxonomies  did 
not  map  directly  on  team  performance  in  command  and  control  teams.  For  example,  communication  and 
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coordination  are  often  used  as  positive  indicators  of  teamwork.  However,  in  C3  teams,  the  environment 
constrains  communication,  and  more  effective  teams  communicate  less  often  and  coordinate  in  more 
anticipatory  ways.  This  reliance  on  information-push  rather  than  information  pull,  or  implicit  coordination, 
has  been  related  to  C3  team  effectiveness  in  more  controlled  settings  (Serfaty  et  al.,  1998). 

Thus,  while  existing  taxonomies  were  very  useful,  we  began  instead  by  considering  the  core 
definition  of  teams.  Teams  are  distinguished  from  groups  in  general  by  a  common  purpose  or  goal, 
performed  by  interdependent  team  members  (Salas  et  al.,  1992).  From  this  definition  we  can  then  derive 
a  sub-definition,  of  teamwork  per  se,  in  that  the  fundamental  function  of  teamwork  can  be  defined  as  the 
effective  managing  of  interdependencies  to  accomplish  a  team.  The  importance  of  considering  type  and 
degree  of  interdependence  when  characterizing  teams  has  been  discussed  (Saavedra,  Earley,  &  Van 
Dyne,  1993;  Salas,  et  al.,  1992).  Rather  than  explicating  behaviors  that  may  (or  may  not)  be  related  to 
the  management  of  interdependencies,  we  then  focused  on  identification  of  the  nature  of  these 
interdependencies.  We  started  with  six  functional  dimensions  of  team  interdependence  which  could  be 
referred  to  in  distinguishing  types  of  teams  (Schiflett  &  Elliott,  in  review).  This  framework  was  later 
refined,  particularly  to  distinguish  coordination  which  was  based  on  task  execution  of  a  plan  (however 
complex)  from  the  construct  of  dynamic  replanning  (Elliott,  Hollenbeck,  Schiflett,  &  Dalrymple,  in  review). 

Our  generic  taxonomy  serves  as  an  overarching  framework-  to  distinguish  teams  in  general, 
across  performance  domains,  with  regard  to  the  nature  and  extent  of  teamwork  demands.  Assuming  the 
■  essence  of  teamwork  is  the  managing  of  team  member  interdependencies,  we  identified  two  core 
dimensions  with  regard  to  nature  and  extent  of  interdependencies,  that  of  (a)  coordination  complexity 
(static)  and  (b)  adaptive  replanning  (dynamic).  A  brief  description  follows. 

Coordination  Complexity.  Assuming  interdependencies  exist,  teams  will  vary  in  the  degree  to 
which  these  interdependencies  can  be  articulated  in  advance.  For  example  a  symphony  can  explicate  its 
interdependencies  among  team  members  to  very  specific  terms,  by  articulating  synchronized  procedural 
strategies  (who  must  do  what,  with  whom,  and  when),  given  a  score  of  music. 

Assuming  the  coordination  plan  is  explicit,  quantitative  delineation  of  the  coordination  flow  would 
yield  an  estimate  of  coordination  complexity.  Evaluation  of  team  coordination  can  then  be  based  on 
adherence  to  the  plan  to  provide  a  measure  of  coordination  efficiency.  Many  approaches  exist  which  can 
provide  indices  of  both  coordination  efficiency  (of  the  plan)  and  coordination  performance,  as  determined 
by  execution  of  the  plan  (Bowers,  Baker,  &  Salas,  1994;  Coovert,  &  McNelis,1992;  Coovert  &  Craiger, 
1997,  Coovert,  Craiger,  Cannon-Bowers,  Driskell,  &  Salas,  1995).  Entire  disciplines  have  focused  on  the 
quantification  of  efficiency.  Highly  sophisticated  mathematical  modeling  techniques  are  arising  from 
numerous  quantitative  disciplines  and  are  becoming  increasingly  user-friendly  through  advances  in  PC 
technology.  Many  of  these  models  have  been  applied  to  the  performance  domain  of  command  and 
control,  to  ascertain  the  effectiveness  of  various  team  task  architectures  and/or  to  generate  more  effective 
structures  (Benson,  Kemple,  Kleinman,  Porter,  &  Serfaty,  1998;  Brehmer,  1992, 1998;  Curry,  Kleinman,  & 
Pattipati,  1998;  Kleinman,  1998;  Krackhardt  &  Carley,  1998;  Handley,  Zaidi,  &  Levis,  1998;  Lee  &  Carley, 
1998;  Levchuk,  Pattipatti,  &  Krackhardt  &  Carley,  1998;  Levchuk,  Pattipati,  &  Kleinman,  1998;  Levchuk! 
Lou,  Levchuk,  Pattipati,  &  Kleinman,  1999;  Serfaty,  1996;  Paley,  Levchuk,  Serfaty,  &  MacMillan,  1999). 
Thus,  the  opportunity  exists  to  model  and  evaluate  the  efficiency  and  execution  of  coordination  plans  of 
very  complex  interactions. 

In  the  modeling  of  complex  team  performance,  it  is  necessary  to  consider  and  evaluate  the 
allocation  of  resources  as  well  as  the  allocation  of  tasks  among  team  members.  For  example,  given  an 
overall  task  goal,  one  usually  finds  a  sequence  of  tasks  which  must  be  accomplished,  resources  for  task 
accomplishment,  and  individuals  to  perform  the  tasks.  To  generate  a  coordination  plan,  tasks  must  be 
assigned  to  resources,  and  to  individuals,  in  some  (hopefully)  efficient  and  effective  manner.  Factors  to 
consider  include  (a)  allocation  of  tasks  to  team  members  (simple  to  complex);  (b)  allocation  of  resources 
to  team  members  (simple  to  complex);  (c)  degree  of  performance  monitoring  (implicit  coordination 
requirement);  and  (d)  degree  of  information  exchange  (explicit  coordination  requirement). 
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Our  fundamental  vision  for  a  model  of  coordination  complexity  represents  the  degree  to  which 
team  members  must  manage  prespecified  interdependencies  with  regard  to  tasks  and  resources  in  order 
to  accomplish  their  mission.  Measurement  would  reflect  the  degree  of  complexity  in  terms  of  coordination 
demand. 

Adaptive  Dynamic  Replanning.  A  second  independent  dimension  of  teamwork  regards  the 
degree  to  which  team  members  must  adapt  to  dynamic  circumstances.  Teams  may  be  very  low  on 
coordination  complexity  as  defined  by  the  complexity  of  a  prespecified  plan,  and  yet  still  have  a  high 
requirement  for  adaptive  teamwork — to  rework  the  plan,  and  re-allocate  tasks  and/or  resources.  Given  a 
dynamic  environment,  teams  must  be  able  to  problem  solve  and  replan  “on  the  fly.”  It  then  becomes 
difficult,  and  perhaps  unfair,  to  evaluate  the  team  through  adherence  to  a  prespecified  coordination  of 
action.  Teams  in  general  can  be  distinguished  with  regard  to  the  degree  to  which  the  team/leader  must 
generate  coordination  strategies  during  performance  execution  as  opposed  to  following  a  predetermined 
and  static  plan.  Certainly,  execution  of  a  complex  reciprocal  coordination  plan  can  be  challenging  even  if 
determined  beforehand,  but  the  task  requirement  is  different  when  plans  must  be  revised  or  improvised 
during  performance  execution.  The  contrast  is  obvious  between  orchestra  versus  jazz  bands,  or  theatre 
acts  versus  performance  art.  However,  most  teams  will  vary  in  more  moderate  levels  for  both  static  and 
dynamic  aspects  of  coordination.  For  example,  most  sports  teams  have  a  repertoire  of  “strategies,”  and 
also  a  set  of  contingencies  for  switching  strategies  if  the  current  strategy  is  not  working  well. 

In  AWACS  teams,  there  can  be  high  levels  of  both  coordination  complexity  and  adaptive 
replanning.  This  is  reflected  in  the  importance  of  premission  planning  where  team  members  are 
informed  of  the  mission  goals,  strategies,  and  constraints.  The  team  leader  then  facilitates  planning 
operational  procedures,  generating  a  communication  protocol,  and  explicating  contingencies  with  the 
other  team  members.  They  also  coordinate  with  other  entities  (e.g.,  pilots)  with  whom  they  must  interact 
during  mission  execution,  ascertaining  rules  and  terms  to  facilitate  and  clarify  later  communications.  This 
planning  phase  is  critical  when  scenarios  are  complex,  demanding,  and/or  likely  to  change.  During  this 
time,  team  members  assign  responsibilities  and  formulate  contingency  plans  for  unexpected  events. 
Thus,  while  most  of  the  planning  is  performed  in  advance,  there  may  be  need  for  dynamic  replanning, 
particularly  if  the  planning  was  not  performed  well  to  start,  and/or  unexpected  events  arise. 

Discussion  with  focal  groups  comprised  of  AWACS  subject-matter  experts  led  to  further 
refinement  of  our  team  task  taxonomy  as  it  applies  to  AWACS  team  performance.  Prior  to  the 
ROADRUNNER  '98  exercise,  we  interviewed  38  expert  AWACS  weapons  directors.  These  groups  were 
interviewed  to  provide  feedback  and  refine  the  teamwork  constructs,  the  DMT  mission  scenarios,  and  the 
team  observer  evaluation  form.  Review  of  the  initial  taxonomy,  which  was  generated  to  classify  teams  in 
general,  led  to  the  identification  of  four  teamwork  functions  which  are  specific  to  AWACS  teamwork:  (a) 
mission  planning,  (b)  communication  content/timing,  (c)  adherence  to  communication  protocol,  and  (d) 
dynamic  replanning  (see  Figure  1). 
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Figure  1.  Core  Functions  of  AWACS  Teamwork 


Mission  planning:  formulation  of  “contracts.”  AWACS  performance  can  be  divided  into  three 
phases:  premission  planning,  task  execution,  and  debriefing  (discussion  after  task  execution).  Mission 
planning  affects  AWACS  teamwork  by  establishing  roles,  responsibilities,  and  contingency  plans  As 
discussed  by  Fahey  et  al.  (1998)  and  Macmillan  et  al.  (1998)  AWACS  weapons  directors  explicate  roles, 
responsibilities,  and  strategies  to  manage  team  member  interdependencies  through  the  establishment  of 
^contracts.  These  contracts  are  made  among  AWACS  team  members  (internal  contracts)  and  to  the 
external  team  (i.e.,  pilots  of  friendly  assets).  In  addition,  WD  focus  groups  strongly  endorsed  the 
importance  of  this  construct. 

Planning  may  greatly  influence  the  flexibility  of  teams  during  performance  through  two 
mechanisms  preliminary  task  allocation  (delineation  of  objectives,  roles,  and  responsibilities)  and 
explication  of  contingencies.  First,  the  task  responsibilities  of  the  AWACS  WD  are  articulated  during  the 
premission  briefing.  Responsibilities  may  be  distributed  geographically,  such  that  WDs  are  performing 
more  like  generalists,  each  performing  similar  broad  duties  but  responsible  for  a  different  geographical 
area.  However,  they  may  also  be  distributed  functionally,  so  that  one  WD  may  be  responsible  for 
“offense,”  another  for  “defense,”  and  another  for  refueling  of  assets,  and  all  work  across  the  entire 
geographic  area.  In  this  functional  configuration,  it  is  easier  for  each  member  to  maintain  awareness  of 
the  entire  battlespace,  thus  increasing  shared  awareness,  but  coordination  demands  among  members 

are  higher.  The  manner  of  task  allocation  thus  directly  affects  both  coordination  complexity  and  adaDtive 
flexibility. 

A  second  way  planning  affects  adaptive  performance  is  through  explication  of  contracts  and 
contingencies.  During  the  premission  briefing,  “contracts”  are  formed  among  the  team  members,  which 
assign  and  describe  roles  and  responsibilities.  These  contracts  include  the  task  allocations,  but  also 
address  additional  details  and  contingencies,  such  as  review  of  the  flow  of  aircraft  assigned  to  each  WD 
(ingress/egress,  sequence,  timing,  role,  aircraft  positioning),  expected  enemy  tactics  and  anticipated/  pre¬ 
briefed  actions  by  friendly  aircraft,  the  assignment  of  communication  channels,  and  scope  setup 
assignments.  Scope  setup  includes  issues  like  who  will  initiate  update  of  display  features  such  as  track 
initiations,  special  points,  restricted  areas,  etc.;  how  /  what  will  be  displayed  during  the  different  phases  of 
the  mission  (e.g.,  the  primary  controller  may  have  minimum  information  displayed  and  be  concentrating 
on  radar  data  while  another  may  have  a  more  cluttered  display);  and  contingencies  for  scope  problems. 
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In  the  communications  setup,  planning  typically  addresses  how  communications  will  be  configured  and 
identifies  the  responsibilities  for  each  person  during  each  phase  of  the  mission— such  as  who  will 
monitor/talk  on  specific  radio  frequencies,  what  is  the  priority  of  information  to  be  passed  from  assist  to 
the  primary  WD,  the  radio  plan,  and  the  format  for  passing  information  from  assist  to  primary  (for 
example,  point  and  talk,  time  share  frequency,  or  grab-arm  technique).  Emergency  contingencies  are 
also  discussed  and  roles/responsibilities  assigned.  These  reflect  some  of  the  many  details  included  in 
well-formulated  contracts.  There  are  numerous  additional  details  that  can  be  discussed;  one 
characteristic  of  a  good  mission  planner  is  the  ability  to  recognize,  given  a  particular  mission,  which 
details  should  be  emphasized. 

Communication  discipline:  adherence  to  protocol.  AWACS  weapons  director  tasks  are 
based  predominately  on  the  exchange  of  verbal  information.  Communications  are  heavily  standardized  in 
terms  of  content  (jargon)  and  process.  This  aspect  of  communication  effectiveness  refers  to  the  degree 
to  which  individuals  follow  guidelines  for  communication  exchange.  In  addition  to  proper  jargon  and 
syntax,  communications  must  be  clear,  concise,  and  correct.  This  construct  was  identified  in  all  cognitive 
task  analyses  and  by  the  focal  groups,  and  is  an  explicit  and  important  aspect  of  WD  training  that  applies 
across  all  mission  scenarios. 

Communication  for  shared  situation  awareness:  content,  timing.  Communications  may  be 
clear,  concise,  correct,  and  delivered  according  to  proper  protocol  and  still  be  superfluous.  The 
maintenance  of  shared  situation  awareness  also  requires  that  pertinent  information  be  exchanged  with 
the  right  person,  at  the  right  time.  In  such  a  communication-rich  environment,  too  much  communication 
can  impede  performance  when  unnecessary  information  “steps  over”  other,  more  urgent  communication. 
Indeed,  part  of  the  proper  timing  of  AWACS  communications  involves  knowledge  of  when  to  keep  quiet, 
particularly  when  communicating  to  pilots. 

Dynamic  Replanning.  This  construct  started  off  as  supportive  behavior,  as  suggested  by 
existing  taxonomies  of  team  performance.  However,  while  it  would  seem  apparent  that  supportive 
behavior  would  be  relevant,  the  conceptualization  of  supporting  behavior  among  AWACS  weapons 
directors  was  difficult  to  validate  through  SMEs.  While  there  was  no  doubt  that  AWACS  team  members 
do  support  each  other,  the  argument  was  made  that  most  of  the  support  is  in  the  form  of  communication 
exchange,  already  captured  as  constructs.  -AWACS  team  members  support  each  other  primarily  through 
updates  and  reminders  of  salient  information. 

We  then  discussed  the  nature  of  dynamic  replanning  and  its  relevance  to  AWACS  team 
performance;  this  was  very  highly  endorsed.  In  fact,  team  members  appreciated  the  more  academic 
nature  of  the  “dynamic  replanning”  label,  as  compared  to  their  own  references,  which  were  more 
interesting  and  colorful.  The  common  factor  here  is  the  need  to  recognize  when  plans  are  no  longer 
effective,  generate  alternatives,  and  suggest/decide  upon  a  new  course  of  action. 

It  should  be  noted  here  that  these  constructs  are  meant  to  be  core  functions  that  comprise 
AWACS  teamwork.  There  are  likely  to  be  additional  behaviors,  such  as  the  monitoring  of  other  team 
member  performance,  which  would  also  be  related  to  teamwork.  However  those  behaviors  are  predicted 
to  affect  teamwork  through  their  affect  on  one  of  these  functions.  Thus,  monitoring  other  team  members 
would  be  expected  to  affect  the  degree  to  which  communications  support  shared  situation  awareness,  or 
the  timeliness  of  problem  recognition.  Also,  additional  behaviors  may  be  more  dependent  on  mission 
scenario.  Monitoring  of  others  may  be  critical  for  teamwork  in  one  scenario,  yet  unnecessary  for  another. 
We  wanted  to  ascertain  the  core  functions  that  are  demanded  across  scenarios. 

Development  of  Rating  Instrument 

Once  the  dimensions  of  AWACS  teamwork  performance  were  refined  by  the  SME  focus  groups, 
the  rating  instrument  was  tailored  to  capture  these  dimensions  across  three  phases  of  performance:  (a) 
premission  planning,  (b)  mission  performance,  and  (c)  mission  debriefing.  Ratings  were  based  on 
traditional  rating  assessment  categories.  The  rating  categories  used  are  based  on  the  same  categories 
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of  performance  used  in  AWACS  WD  training.  A  description  of  the  rating  scale  and  evaluation  form  is 
provided  in  the  appendix. 

Premission  planning.  WDs  were  rated  on  the  following  during  the  planning  phase  of  their 
performance.  The  development  of  mission  aids  including  the  generation,  refinement,  and  review  of  the 
communications  worksheet  (which  specifies  who  talks  to  whom  on  which  channel),  the  fighter  flow  sheet, 
the  chart,  and  the  fact  sheet,  all  of  which  describe  and/or  specify  procedures  to  enhance  awareness  and 
coordination.  Formulation  of  internal  contracts  refers  to  the  agreements  made  among  the  WDs  as  to  their 
roles  and  responsibilities.  Communication  tasks  and  console  assignments  are  set,  specific  mission 
objectives  are  discussed,  and  functional  responsibilities  and  contingency  plans  are  specified  and 
assigned.  Formulation  of  external  contracts  refers  to  the  procedures  executed  between  the  WDs  and 
others  (pilots)  with  regard  to  roles  and  responsibilities,  such  as  those  regarding  objectives,  strategies,  and 
communications.  These  contracts  are  then  discussed  within  the  briefing  session  by  the  lead  WD  with  the 
lead  pilot,  along  with  issues  such  as  the  rules  of  engagement  (ROE)  and  ATO  compliance. 

Mission  Execution.  Performance  during  mission  execution  was  assessed  by  ratings  regarding 
(a)  communication  in  accordance  with  Air  Force  Tactical  Publication  and  Unit  Standards,  (b) 
communications  in  support  of  situation  awareness  and  the  “big  picture”,  and  (c)  overall  mission  execution, 
as  specified  during  premission  planning  and  adaptive  problem  solving.  Communications  are  assessed  in 
terms  of  adherence  to  established  protocol,  which  strives  to  maximize  aspects  of  clarity,  brevity,  and 
accuracy — communications  that  are  clear,  concise,  and  correct.  This  includes  communications  among 
WDs  (labelled  “internal”)  and  also  their  communications  to  others  (labelled  “external”).  In  addition, 
communications  are  also  rated  with  regard  to  relevance  and  timeliness— is  the  information  what  was 
needed  to  maintain  team  situation  awareness — was  the  right  information  “pushed”  to  the  right  person  at 
the  right  time?  Communication  for  situation  awareness  was  also  rated  for  within-team  (internal)  and  to 
others  (external).  Mission  execution  was  assessed  through  consideration  of  ATO  execution,  contract 
execution  (formulated  during  premission  briefing),  and  adaptive  replanning  (flexibility,  problem-solving, 
contingency  generation  as  needed). 

Mission  Debriefing.  Performance  during  mission  debriefing  was  assessed  through  ratings  of  (a) 
reconstruction  of  the  engagement,  (b)  evaluation  of  team  objectives,  (c)  review  of  equipment  issues,  (d) 
review  of  team  mission  execution,  and  (e)  review  of  communications.  Reconstruction  of  the  engagement 
was  assessed  through  consideration  of  the  process  of  reviewing  the  recordings  and  the  identification  of 
conflicts  and/or  problems.  Objectives  were  reviewed  in  light  of  mission  support  and  execution  of 
contracts.  Equipment  issues  included  communication,  console  assignments,  and  any  failures/alibis. 
Team  mission  execution  was  assessed  through  consideration  of  priority  objectives,  training  objectives, 
and  any  failunes/alibis.  Communications  was  considered  with  regard  to  within-team  (internal)  information 
exchange  and  also  communications  to  others  (external).  Communications  should  have  supported  overall 
situation  awareness,  and  lessons  learned  should  be  identified  during  the  mission  debriefing. 

Each  team  was  rated  on  the  various  aspects  of  performance,  described  previously,  within  each 
of  three  phases  of  performance,  for  each  mission.  They  were  assessed  for  performance  in  Phase  I 
(premission  briefing),  Phase  II  (mission  execution),  and  Phase  III  (mission  debriefing). 

Results 

Descriptive  Statistics.  Table  1  provides  mean  performance  ratings  across  teams  and 
scenarios.  Results  indicate  a  good  amount  of  variance  in  distribution  in  ratings,  with  utilization  of  the 
entire  rating  scale,  for  most  measures.  This  indicates  the  scenarios  were  neither  too  difficult  nor  too 
easy,  but  were  sufficiently  challenging  to  distinguish  team  member’s  performance.  It  also  indicates  that 
the  measures  chosen  were  aspects  of  performance  that  differed  among  team  members.  We  did  not  have 
measures  where  all  team  members  excelled  or  failed.  Of  interest  were  relationships  among  measures, 
and  the  degree  to  which  measures  were  affected  by  (a)  the  rater  (source  of  error),  (b)  the  team  (how 
different  were  the  teams  in  their  performance  and  were  there  particular  aspects  that  most  differentiate 
them,  and  (c)  experience  over  time. 
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Table  1.  Average  Ratings  across  Teams,  Scenarios,  Raters 


Premission  Planning 

N 

Min 

Max 

Mean 

Std.  Dev 

Mission  Aids 

25 

2.00 

4.00 

3.0800 

.4933 

Internal  Contracts 

25 

2.00 

4.00 

2.9600 

.6758 

External  Contracts 

24 

1.50 

4.00 

2.8125 

.6726 

Prebrief  with  Pilots 

24 

1.00 

4.00 

2.8542 

.6833 

Planning  Overall 

25 

1.88 

3.75 

2.9200 

.4703 

Task  Execution 

Internal  Communications 

42 

1.00 

4.00 

2.9881 

.7690 

External  Communications 

42 

1.00 

4.00 

2.7143 

.7741 

Sit  Awareness  (Internal) 

42 

1.00 

4.00 

2.9524 

.7949 

Sit  Awareness  (External) 

42 

1.00 

4.00 

2.6905 

.8762 

Mission  Execution  (Internal) 

42 

1.00 

4.00 

3.0000 

.7730 

Mission  Execution  (External) 

41 

1.00 

4.00 

2.6951 

.9413 

Overall  Mission  Execution 

42 

1.33 

4.00 

2.8421 

.7301 

Debrief 

Reconstruction  of  Events 

24 

2.00 

4.00 

3.1250 

.6124 

Discussion  of  Equipment 

26 

2.00 

4.00 

3.1154 

.4961 

Communications  (External) 

25 

1.00 

4.00 

3.0400 

.8406 

Communications  (Internal) 

24 

1.00 

4.00 

2.9792 

.8140 

Discuss.  Team  Performance 

24 

2.00 

4.00 

3.0417 

.7929 

Discuss,  of  Mission  Outcome 

24 

1.00 

4.00 

3.0833 

.8805 

Overall  Debrief 

27 

1.50 

4.00 

3.0562 

.5970 

OVERALL  across  3  phases 

22 

1.94 

3.69 

2.8699 

.4705 

Rater  Error.  Analysis  of  Variance  (ANOVA)  indicated  no  significant  differences  among  raters 
with  regard  to  any  of  the  measures.  In  addition,  regression  analyses  based  on  the  consideration  of 
trusted  agents,  teams,  and  scenarios  indicate  that  the  variance  attributable  to  differences  among  trusted 
agents  was  not  significant  (see  Table  2). 


Table  2.  Overall  Regression  Model  Summary:  Raters,  Teams,  Scenario  Experience  (Training) 


Model  R  R  Square  Adjusted  R  Square  Std.  Error  of  the  Estimate 


.859 

.738 

Sum  of  Squares 

Regression 

3.429 

Residual 

1.220 

Total 

4.648 

(Constant) 

t 

11.750 

TRUSTEDA 

-.969 

TEAM 

-2.436 

SCENARIO 

6.557 

.694  .2603 

df  Mean  Square  F  Sig. 

3  1.143  16.867  .000 

18  6.776E-02 

21 

Sig. 

.000 

.345 

.025 

.000 


Relative  impact  of  raters,  teams,  and  scenarios.  There  were  three  primary  sources  of 
variance:  that  attributable  to  raters  (which  should  be  minimized),  to  differences  among  teams  (not  of 
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primary  interest  here),  and  to  the  training  exercise  itself  (differences  among  the  seven  scenarios  were 
predicted  to  demonstrate  improvement  over  time).  Multiple  regression  analyses  were  performed,  first  on 
the  model  composed  of  trusted  agents,  teams,  and  scenarios.  This  model  predicted  73%  of  the  total 
variance  in  ratings.  In  addition,  the  consideration  of  trusted  agents  did  not  add  to  the  prediction  of  ratings, 
thus  indicating  reliability  of  measures  with  regard  to  raters 

These  data  indicate  that  performance  improved  across  all  teams  for  most  measures  of 
performance  over  the  three  phases  of  performance.  Results  are  described  in  further  detail  for  the  effects 
due  to  differences  among  teams  and  differences  across  scenarios. 

Differences  among  teams.  Data  demonstrate  the  advantage  in  careful  development  of 
performance  constructs.  The  overall  measure  of  performance  was  based  on  the  mean  of  all  measures; 
ANOVA  results  did  not  indicate  significant  differences  among  teams.  There  were,  however  differences 
among  teams  for  the  measurement  of  one  aspect  (negotiation  of  contracts  among  team  members)  of 
performance  during  premission  briefing,  and  for  nearly  all  aspects  of  performance  during  mission 
execution.  Teams  did  not  differ  significantly  in  performance  during  mission  debriefing.  An  overall 
assessment  of  performance  may  be  too  generic  to  capture  effects  of  a  predictor  variable. 


Table  3:  Performance  Ratings  with  Significant  Differences  Between  Teams 

ANOVA _ _ _ 


Variable 

Sum  of  Squares 

df 

Mn  Sq 

F 

Sig. 

Planning:  Internal  Contracts 

3.363 

2 

1.681 

4.869 

.018** 

Mission  Execution-External 

5.730 

2 

2.865 

3.665 

.035** 

Sit.  Awareness-External 

4.298 

2 

2.149 

3.083 

.057* 

Sit.  Awareness-Internal 

4.762 

2 

2.381 

4.392 

.019** 

Communications-External 

3.857 

2 

1.929 

3.631 

.036** 

Communications-lnternal 

2.869 

2 

1.435 

2.617 

.086* 

Overall  Mission  Execution 

3.298 

2 

1.649 

3.466 

.041** 

|Mission 

Aids 

|  External 
Contracts 
|  Internal 
Contracts 

Iprebrief- 

Pilots 

Overall 

Planning 


Figure  2.  Performance  of  Teams  in  Mission  Planning 
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Figure  3.  Performance  of  Teams  in  Mission  Execution 


Impact  of  training  experience.  Team  performance  differed  significantly  across  scenarios  for  many  of  the 
measures  (See  Appendix  for  descriptives  and  ANOVA  output).  Differences  in  measures  among 
scenarios  would  capture  differences  due  to  the  scenario  content,  as  scenarios  differed  in  mission 
objective  and  content.  They  would  also  be  due  to  experience  over  time.  Scenario  1  and  Scenario  7  were 
developed  to  be  very  similar  in  mission  and  task  demands,  to  better  ascertain  the  effect  of  experience  on 
performance.  The  following  graph  (fig.  5)  describes  overall  results  of  instructor’ratings  of  AWACS  team 
performance,  for  each  mission  session,  ordered  overtime. 


4.0 

3.5 

3.0 

2.5 

S  2.0 

1.5 

1.0 

.5 
0.0 

1  2  3  4  5  6  7 

SCENARIO 


Figure  5.  Averaged  Team  Performance  for  each  Phase  and  Scenario,  Over  Time. 


Figure  5  provides  a  succinct  glimpse  of  trends  regarding  the  effect  of  simulation  participation  on 
performance,  through  differences  in  performance  from  Scenario  1  and  Scenario  7,  by  phase.  Here  it  can 
be  seen  that  variance  in  performance  across  scenarios  was  more  attributable  to  performance  in  Mission 
Execution  and  Debriefing:  teams  were  lower  in  performance  during  these  phases  during  the  first  scenario, 
and  demonstrated  higher  improvement  in  these  phases.  At  the  same  time,  performance  during 
premission  briefing  also  improved  over  time. 

Team  1  was  the  highest  performing  team  for  both  pre  and  post  scores.  Improvement  for  this 
team  was  consistent  across  the  three  phases.  In  comparison,  Teams  2  and  3  performed  less  well  from 
the  start,  and  particularly  less  well  in  mission  execution  and  debriefing.  Data  indicate  that  training  needs 
exist  for  the  kind  of  training  that  distributed  mission  training  provides:  Improvement  in  team  premission 
briefing,  team  coordination,  and  team  debriefing.  In  addition,  data  demonstrate  the  effectiveness  of 
participation  in  the  training  exercise  for  the  improvement  of  performance  across  all  phases  of  each 
mission. 
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Discussion 


Assessment  of  individual  and  team  performance  in  realistic  combat  training  environments 
requires  (a)  the  capability  to  produce  complex  and  dynamic  scenarios,  (b)  identification  of  constructs  that 
represent  important  individual  and  team  skills,  and  (c)  identification  and/or  development  of  construct 
measures.  Ideally,  scenarios  should  include  embedded  events  that  link  training  requirements, 
performance  assessment,  and  performance  feedback  (Dwyer,  et  al.,  1999).  In  addition,  each  scenario 
must  have  quantitative  criteria  of  mission  success. 

Simulation-driven  distributed  mission  training  was  expected  to  add  value  through  the  experience 
of  complexity  of  team-on-team  interdependencies.  The  immersion  of  individuals  into  teams  within  a  global 
mission  scenario  should  enhance  their  capability  of  managing  these  interdependencies  during 
performance.  For  these  exercises,  the  expanded  scope  of  the  debriefing  experience  was  found  to  be 
particularly  valuable  to  AWACS  crew  as  they  do  not  usually  debrief  with  pilots  after  an  operational 
mission.  An  AWACS  crew  stays  in  flight  well  beyond  the  time  of  performance  of  a  particular  set  of 
combat  aircraft. 

Results  were  consistent  with  expectations.  Teams  demonstrated  significant  improvement  in  several 
aspects  of  performance  of  premission  planning,  mission  execution,  and  mission  debriefing.  In  addition, 
while  measures  of  overall  performance,  based  on  averaged  ratings  were  not  sensitive  to  differences, 
significant  differences  in  performance  were  demonstrated  for  particular  aspects  of  performance,  thus 
indicating  the  advantage  of  careful  construction  of  performance  constructs. 
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